I sometimes see people refer to neural networks as just "another tool in your machine learning toolbox". They have some pros and cons, they work here or there, and sometimes you can use them to win Kaggle competitions. Unfortunately, this interpretation completely misses the forest for the trees. Neural networks are not just another classifier, they represent the beginning of a fundamental shift in how we develop software. The "classical stack" of Software 1.0 is what we're all familiar with -- it is written in languages such as Python, C, etc.

There is a lot of excitement around AI. Recently there has been an incredible amount of buzz around the demos of models like ChatGPT and Dall-E-2. As impressive as these systems are, I think it becomes increasingly important to keep a level head, and not get carried away in a sea of excitement. The following videos/lectures are more focused on how to think about data science projects, and how to attack a problem. I've found these lectures to be highly impactful in my career and enabled me to build effective and practical solutions that fit the exact needs of the companies I've worked for.

The current series of articles on MLOps started with an analogy with DevOps. To explain how software updates happen, I am introducing the Known Unknown matrix. The so-called "known unknown" matrix is popularized by Donald Rumsfeld and divides our knowledge into four quadrants: Applied to software development, let us consider what the knowns and the unknowns for the feature team would be. As an example, let us assume that the software is an OCR program that recognizes car license plate numbers based on heuristic algorithms. In Software 1.0 (the traditional software), bugs and feature requests are two cases when software updates are needed.

In today's world, data has played a huge role in the success of technology giants like Google, Amazon, and Facebook. All of these companies have built massively scalable infrastructure to process data and provide great product experiences for their users. In the last 5 years, we've seen a real emergence of AI as a new technology stack. For example, Facebook built an end-to-end platform called FBLearner that enables an ML Engineer or a Data Scientist build Machine Learning pipelines, run lots of experiments, share model architectures and datasets with team members, scale ML algorithms for billions of Facebook users worldwide. Since its inception, millions of models have been trained on FBLearner and every day these models answer billions of real-time queries to personalize News Feed, show relevant Ads, recommend Friend connections, etc.

Recent blog posts by Andrej Karpathy at Medium.com and Pete Warden at PeteWarden.com have caused a paradigm shift in the way I think about neural nets. Instead of thinking of them as powerful machine learning tools, the authors instead suggest that we should think of neural nets, and in particular, convolution deep nets, as'self-writing programs.' It turns out that a large portion of real-world problems have the property that it is significantly easier to collect the data than to explicitly write the program. A large portion of programmers of tomorrow do not maintain complex software repositories, write intricate programs, or analyze their running times. They collect, clean, manipulate, label, analyze and visualize data that feeds neural networks.

Remember when software was eating the world? The trendy observation these days is that artificial intelligence (AI) is eating software. Even Google CEO Sundar Pichai has talked about software that "automatically writes itself." And certainly if you consider software development to be little more than the creation of oft-repeated segments of code, then the rapid advances in AI would give software engineers pause. Traditionally, developers have written software as a series of hard-coded rules: If X happens then do Y.

I sometimes see people refer to neural networks as just "another tool in your machine learning toolbox". They have some pros and cons, they work here or there, and sometimes you can use them to win Kaggle competitions. Unfortunately, this interpretation completely misses the forest for the trees. Neural networks are not just another classifier, they represent the beginning of a fundamental shift in how we write software. The "classical stack" of Software 1.0 is what we're all familiar with -- it is written in languages such as Python, C, etc. It consists of explicit instructions to the computer written by a programmer.

Andrej Karpathy has an article "Software 2.0" that makes the argument that Neural Networks (or Deep Learning) is a new kind of software. I do agree that there indeed a trend towards "teachable machines" as opposed to the more conventional programmable machines, however I do have an issue with some of the benefits that Karpathy mentions to back-up his thesis. Certainly Deep Learning is already eating the Machine Learning world with advances across the board. Karpathy mentions several well known ones: visual recognition, speech recognition, speech synthesis, machine translation, robotics and games. This frames his argument about the sea change in computing and perhaps its time to think about a new kind of software (I guess the kind that you teach like a dog instead of programming).